Multimodal Interactive User Interfaces for Mobile Multi-Device Environments

Portable devices come with different individual limitations in user interaction like limited display size, small keyboard and different sorts of input and output channels. To allow ubiquitous, peripheral interaction, a flexible way to provide user interfaces for multiple devices and their different modalities is needed. However, current systems and formats do not sufficiently integrate advanced multimodal interaction and multiple devices. In that context, we present our approach on an advanced system for Multimodal Interaction and Rendering (MIRS) tackling three angles of the problem: UI description, selection of best device and modality, and the UI transformation/adaptation to the selected device. While the UI description is handled by our newly developed XML-based Dialog and Interface Specification Language (DISL), the deviceand modality selection employs a profile-based approach, whereas the advanced UI transformations are built upun rule-based transcoding techniques.